One comment regarding TCP (and UDP) Dynamics, a standard internet packet (MTU) is about 1500 bytes. A single 4 KB chunk already requires 3 TCP packets. Whether you flush chunks to the OS individually or in a batch, the OS TCP stack will still stream them out in 1500-byte packets at the maximum rate the network allows.

Until the real cluster measurements are performed, I have run a benchmarks on local machine, but using dnctl to simulate the latency on the network interface.

The benchmark that is executed is:

go test -bench=BenchmarkDeliveryLibP2P/n=120 -benchmem -benchtime=5x ./pkg/pullsync

Results without added latency:

goos: darwin
goarch: arm64
pkg: github.com/ethersphere/bee/v2/pkg/pullsync
cpu: Apple M1 Pro
BenchmarkDeliveryLibP2P/n=120/Batched-10               5         387043375 ns/op         2456576 B/op      27140 allocs/op
BenchmarkDeliveryLibP2P/n=120/v2.8.1-10                5         388048475 ns/op         1734056 B/op      27794 allocs/op
PASS

Than a latency is set with:

sudo dnctl pipe 1 config delay 200ms
echo "dummynet out from 127.0.0.1 to 127.0.0.1 pipe 1" | sudo pfctl -f - -e

And the benchmark is run again, the results are:

goos: darwin
goarch: arm64
pkg: github.com/ethersphere/bee/v2/pkg/pullsync
cpu: Apple M1 Pro
BenchmarkDeliveryLibP2P/n=120/Batched-10               5        1622241875 ns/op         2409558 B/op      27160 allocs/op
BenchmarkDeliveryLibP2P/n=120/v2.8.1-10                5        1625406400 ns/op         1730278 B/op      27831 allocs/op
PASS

If you want to reproduce the benchmarks on the local machine, do not forget to remove the latency:

sudo pfctl -F all -d
sudo dnctl -q flush

The results show that the latency has an impact on the performance of the pullsync algorithm, between two runs (with and without latency), but there are no differences between the batched and unbatched version of the algorithm. he latency penalty in pullsync is a TCP/IP layer phenomenon governed by the initial handshake and bandwidth delay, not an application-level delivery issue.

Both implementations suffer from the exact same latency penalty (TCP Slow Start): When a 200ms delay is injected, both versions see their delivery time increase from ~0.38s to ~1.62s. This ~1.2-second penalty is caused by the initial protocol handshake (the Get -> Offer -> Want) and the TCP congestion window (slow start) ramping up across a high-latency link.

Batching application-level data does not improve stream performance: With 200ms of latency, the Batched implementation (1.622s) and the v2.8.1 implementation (1.625s) are essentially identical. This proves that because legacy pullsync already streams data into the socket buffer rather than waiting for individual application-level ACKs, application-level batching provides no benefit. In both cases, the OS segments the stream into identical 1500-byte MTU packets and transmits them at the exact same rate.

While providing no speed benefit, the Batched implementation consistently consumes more memory (2.4 MB/op) compared to the legacy v2.8.1 implementation (1.7 MB/op). This is because the batched version must allocate and hold large arrays of chunk data in memory simultaneously before writing them to the socket buffer.

In this test, the batching protocol change introduces memory overhead without delivering any network speedup compared to the bee v2.8.1 pullsync implementation.

Measurements on the light testnet is performed by @gacevicljubisa and me.

The setup is to have only two nodes in the network running but not connecting to any other nodes, not even to themselves. One node is the uploader node and another node is the syncing node that share the same neighbourhood 0000000 (by using --target-neighborhood cli option on setup). First, a random file of 100MB length is uploaded to the uploader node (which is previously nuked), when it is done uploading, a connect call is performed to connect the uploader node and syncing node in order for pullsyncing to start. It is monitored that no other nodes are connected to the two nodes in the experiment. The connection after the file has been uploaded is ensuring that pushync is not interfering with the pullsync.

In total of 6 measurements are performed, 3 with bee v2.8.1 and 3 with the batching pullsyncer, implemented in this branch.

Kuberenetes cluster is used for running the bee nodes on light testnet.

The testing script, datasets and html interactive chart are attached to this comment.

The data show high variance in syncing speeds for either of the syncing algorithm. It also show a pattern when every run eventually drops to about 14.5 chunks per second which is most likely doe to disk I/O ceiling on kubernetes cluster. Before the drop, syncing speeds are significantly higher but with variances for both algorithms in the same ranges.

If we look at the pure transfer speed (the first 10,000 chunks) before the IO limits kick in:

• Non-Batching (v2.8.1) Speeds: 84, 97, and 156 chunks/sec (Average = 112 chunks/sec)
• Batching Speeds: 156, 89, and 44 chunks/sec (Average = 96 chunks/sec)

Statistically speaking, the variance (the swings between 44 and 156) completely destroys any statistical significance. Both implementations had one run hit ~156 chunks/sec, and both had runs dip below 90. The variance is most likely being driven entirely by the Kubernetes environment (noisy neighbors, CPU allocation, network jitter) rather than the protocol itself.

Because every single run eventually drops down to the ~14.5 chunks per second disk IO ceiling, the total completion time of the sync is almost entirely determined by how long the pod was allowed to burst before hitting the ceiling, rather than how fast the chunks were actually traveling over the network.

Conclusion: due to the extreme environmental noise in the Kubernetes cluster, the data is inconclusive. There is no statistically significant evidence from these specific runs that batching is better.

I would suggest to run the test on the environment with less noise, which wi could prove by having similar results on several runs of the same algorithm. And also to have the environment with local disks.

Chart: pullsync-timeseries.html to be opened in web browser for interactivity, but here is a static image as well:

The testing script: pullsync_benchmark.sh

Data:
pullsync_metrics_batch_1785245830.csv
pullsync_metrics_batch_1785262487.csv
pullsync_metrics_batch_1785273181.csv
pullsync_metrics_v2.8.1_1785243656.csv
pullsync_metrics_v2.8.1_1785275681.csv
pullsync_metrics_v2.8.1_1785277405.csv

CSV files have first column as a timestamp and the third the number of delivered chunks as recorded per metric bee_pullsync_chunks_delivered.

Side note: We have also recorded the pullsyncRate as the third column in the csv files, and its value is completely unreliable. We did not investigate why. The value is calculated on the fly internally in bee and did not interfere with the results, but we did not try to address this issue.